Integration patterns during multimodal interaction

The development of multimodal interfaces and algorithms for multimodal integration requires knowledge of integration patterns that represent how people use multiple modalities. We analyzed multimodal interaction with three different applications. Semantic analysis revealed that multimodal inputs can exhibit cooperation other than complementary and redundancy. Analysis of the relationship between tasks and multimodality revealed a positive correlation between the type of a task and multimodal interaction. In addition, analysis revealed that referential expressions can contain more information than simple deictic terms such as instructions to select or modify the referents.